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I. REAL PARTY IN INTEREST 



The subject application is assigned to Microsoft Corporation, of Redmond 
Washington. 
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II. RELATED APPEALS AND INTERFERENCES 



There are no known related appeals or interferences. 

III. STATUS OF CLAIMS 

Claims 1 through 23 represent all claims currently pending in the application. These 
claims are provided for reference in the attached Appeal Brief Appendix. The rejection of 
claims 1-17 is hereby appealed. No reason for the rejection of Claims 18-23 have ever 
been provided in the prosecution of this case. It is assumed for purposes of this appeal 
that Claims 18-23 are allowed. 

IV. STATUS OF AMENDMENTS 

No amendments are currently pending. 



V. SUMMARY OF THE INVENTION 

In general, the Appellants' claim a system and method that allows for face 
recognition even in the absence of a significant amount of training data. Further, it can 
recognize faces at various pose angles without the need to capture training images 
exhibiting the corresponding pose. This is accomplished by synthesizing training 
images depicting a subject's face at a variety of poses from a small number of actual 
images of the subject's face. (Specification, page 2, lines 7-12) The system and 
process according to the claimed invention employs a generic 3-D graphic face model 
and the small number of images of the subject's face in an automatic deformation 
technique to create a single, specific 3-D face model of the subject from the generic 
model and images. (Specification, page 2, lines 25-30). A subdivision spline surface 



2 



construction technique is next used to "smooth" the specific 3-D face model and a multi- 
direction texture mapping technique is used to endow texture or photometric detail to 
the face model to create a texturized, smoothed, specific, 3-D face model. This 
technique adds realism to the synthetic human faces. (Specification, page 3, lines 18- 
24). 

Once a 3-D face model of a specific subject is obtained, realistic individual virtual 
faces or 2-D face images are synthesized at various poses to create groups of training 
images for input into a "recognizer" to allow for training of the recognizer. It is also 
optionally possible to take the generated images and synthetically vary the illumination 
to produce each image at various illuminations. In this way, subjects can be recognized 
regardless of the illumination characteristics associated with an input image. 
(Specification, page 4, lines 9-17). 

VI. ISSUES 

In the final Office Action dated March 5, 2004, claims 1-1 1 and 16 were rejected 
under 35 USC 103(a) as being unpatentable over Yan et al's publication, in view of Kung, 
U.S. Patent No. 5,850,470. Claims 12-15 were rejected under 35 USC 103(a) as 
unpatentable over Yan, in view of Kung, and in further view of Deering, U.S. Patent No. 
6,525,723, hereinafter Deering. Claim 17 was rejected under 35 USC 103(a) as 
unpatentable over Yan, in view of Kung and in further view of Georghiades, June 1999. 
The Appellants respectfully disagree with this contention of obviousness. 

VII. GROUPING OF CLAIMS 

1 . Claims 1 -1 1 and 1 6 stand or fall together. 

2. Claims 1 2-1 5 stand or fall together 

3. Claim 17 stands or falls alone. 
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VIII. THE EXAMINER'S RATIONALE 



The Examiner's rationale for the rejection of independent claim 1 under 35 
U.S.C. §103 as being obvious by Yan et al's publication, in view of Kung, U.S. Patent 
No. 5,850,470 was stated in the Final Office Action of January 12, 2004 and maintained 
in the Advisory Action to the Appellants after final Response is as follows: 

"I. Applicant argues that reference Yan fails to disclose the following 
features:.... b) training of a 3D face recognizer, (page 12, paragraph 2).... With 
regards to the limitation b) the limitation is taught by the teaching reference Kung, 
see col. 4, lines 5-13, wherein the lines 10-11 specifically calls for a face database 
16 used to train the face detector. II. Applicant argues that the reference Kung 
fails to disclose the following features: a) no use of a synthesized database to train 
a face recognizer (page 13, paragraph 1)... With regard to the limitation a) the 
limitation is disclose by the primary reference Yan, synthesizing various face pose 
images, see page 857, col.2, lines 17-32, the images are synthesized and calls for 
using the manipulating polygonal mesh of human face model and further smoothing 
the images using the plastic visco elastic behavior of the facial skin i.e. "synthesizing 
various face pose images."... Arguments regarding Claims 12-15 and 17... Applicant 
argues that Yan, Kung and Deering fail to disclose a) a training recognition system 
that recognizes a person based on a single image by employing this image to 
generate a synthesized database (page 15 and 16 paragraph 2)... The claim 1, 
claims the limitation of 'various face pose images" i.e. more then one or an image 
that is disclosed by the reference Yan see figure 6, wherein three poses of a subject 
is taken and synthesized." 



IX. ARGUMENT 
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A. The 35 U.S.C. S103(e) Rejection of Claims 1-11 and 16: 



As explained by the Appellants 1 in their response filed August 25, 2003, and again in 
the Appellants' Request for Reconsideration filed November 17, 2003, the Office Action 
rejected independent claim 1 under 35 U.S.C. §1 03(e), based on the rationale that Yan 
discloses a deformable model of a realistic face but does not teach the training of a 3D 
face recognizer for face recognition. However, the Office Action further contended that 
Kung teaches a facial recognition system using a database of images to train the 
recognizer. The Office Action argued that this combination teaches the Appellants 
invention. The Appellants respectfully disagree with this contention of obviousness. 

In order to deem the appellants 1 claimed invention unpatentable under 35 USC 
§103, a prima facie showing of obviousness must be made. To make a prima facie 
showing of obviousness, all of the claimed elements of an appellant's invention must be 
considered, especially when they are missing from the prior art. If a claimed element is not 
taught in the prior art and has advantages not appreciated by the prior art, then no prima 
facie case of obviousness exists. The Federal Circuit court has stated that it was error not 
to distinguish claims over a combination of prior art references where a material limitation 
in the claimed system and its purpose was not taught therein (In Re Fine, 837 F.2d 107, 5 
USPQ2d 1596 (Fed. Cir. 1988)). 

The Appellants claim a system and method that allows for face recognition even 
in the absence of a significant amount of training data. Further, it can recognize faces 
at various pose angles without the need to capture training images exhibiting the 
corresponding pose. This is accomplished by synthesizing training images depicting a 
subject's face at a variety of poses from a small number of actual images of the 
subject's face. The system and process according to the claimed invention employs a 
generic 3-D graphic face model and the small number of images of the subject's face in 
an automatic deformation technique to create a single, specific 3-D face model of the 
subject from the generic model and images. Once a 3-D face model of a specific 
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subject is obtained, realistic individual virtual faces or 2-D face images are synthesized 
at various poses to create groups of training images for input into a "recognizer" to allow 
for training of the recognizer. It is also optionally possible to take the generated images 
and synthetically vary the illumination to produce each image at various illuminations. 
In this way, subjects can be recognized regardless of the illumination characteristics 
associated with an input image. 

In contrast, Yan teaches a human face generation technique using a three- 
dimensional deformation technique. The deformation technique allows interactive 
alignment of features in the general geometric face model with the features of the multi- 
direction images of the specific human face which are pre-provided by the animator. 
The deformation result provides an approach to generate 2D images simulating the 
plastic-visco-elastic behavior of the facial skin (page 587, end of second paragraph) 
and exhibiting a facial expression, (page 587, Abstract) Yan requires images in 
several directions to be input in order to apply the skin texture (page 858, last 
paragraph). Yan does not synthesize images of different face poses and, as the first 
Office Action stated, Yan does not teach the training of a 3D face recognizer for 
face recognition. More importantly, the cited Yan reference does not teach the 
synthesis of training images exhibiting different face poses of a subject or the 
training of a 3D face recognizer using synthesized images . 

Although the appellants pointed out this distinction in the last Office Action, the 
Examiner did not respond to this argument of that synthesizing pose-varving training 
images to train a 3D face recognizer is not taught in this cited reference. Yan simply 
does not teach this claimed feature of using synthesizing various face pose images 
using the specific 3-D face model: and employing the synthesized images as 
training images to train a recognizer . Rather Yan teaches creating a specific face 
model for generating images of a subject that exhibit different facial expressions 
(abstract, page 857, page 859, last paragraph), not face poses. It is noted that a face 
pose is a term of art meaning the pitch, roll and yaw angles defining the direction a face 
is pointed usually in relation to a full frontal view. 
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Granted, the Examiner states that Yan teaches this feature at page 857, col. 2, 
lines 6-32 which state, 

"This paper aims at the following two aspects. . . 1 . Based on the 
multi-direction images pre-provided by the animator, we proposed a 
general method for manipulating polygonal mesh of human face model, 
which deform the current face model to get a new face model by 
specifying the new 3-D positions of a set of specific vertexes (feature 
points). When move a vertex to match the specific feature point in the 
image, the deformation algorithm automatically calculate the new position 
of the relative vertex around the moved vertex. This simulate the 
plastic-visco-elastic behavior of the facial skin. 2. Multi-direction 
texture mapping technique is developed to further enhance the realism of 
the synthesized human face. We designed an algorithm to select texture 
information from the frontal view and profile view images of the specific 
subject's face." 

However, this cited passage is just an overview of Yan's process of generating different 
facial expressions by moving vertices of a face model. No mention is made in this 
passage, or the rest of the Yan reference, of generating a set of synthesized 
images to train a recognizer . 

The application of Kung does nothing to change the fact that the cited art 
combination lacks the claimed synthesizing and training features of the 
appellants' claimed invention. Kung discloses a system for automatically detecting 
and recognizing the identity of a deformable object such as a human face, within an 
arbitrary image scene. The system comprises an object detector implemented as a 
probabilistic DBNN, for determining whether the object is within the arbitrary image 
scene and a feature localizer also implemented as a probabilistic DBNN, for 
determining the position of an identifying feature on the object such as the eyes. A 
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feature extractor is coupled to the feature localizer and receives coordinates sent from 
the feature localizer which are indicative of the position of the identifying feature and 
also extracts from the coordinates information relating to other features of the object 
such as the eyebrows and nose, which are used to create a low resolution image of the 
object. A probabilistic DBNN based object recognizer for determining the identity of the 
object receives the low resolution image of the object inputted from the feature extractor 
to identify the object. The system 10 comprises a video camera 12 for inputting an 
arbitrary image scene 1 1 with 320 by 240 pixels. A DBNN-based face detector 14 is 
coupled to the video camera 12 and includes a memory 16 which operates as a 
database for storing images of different human faces. The face detector 14 determines 
whether a face is within the arbitrary image scene 1 1 . The data stored in the face 
database 16 is used to train the face detector 14. During training, updated network 
weighting parameters and thresholds are stored in the face database 16. However, 
Kunq does not teach synthesizing images of different face poses from a specific 
model of a subject and using these synthesized images to create the 
aforementioned database . Rather, Kung teaches extracting a large number of 
training images from videos of a subject. 

Thus, the appellants have claimed elements not taught in the cited art and which 
have advantages not recognized therein, namely the ability to synthesize images of 
different face poses from a specific model of a subject from iust a few actual 
pictures of the subject and using these synthesized images to train a recognizer . 

Accordingly, no prima facie case of obviousness has been established in accordance 
with the holding of In Re Fine. This lack of prima facie showing of obviousness means 
that the rejected claims are patentable under 35 USC 103 over Yan in view of Kung as 
evidenced by the non-obvious claim language of Claim 1 : 

" A computer-implemented process for face recognition, 
comprising using a computer to perform the following process 
actions... inputting an image of a face of a subject sought to be recognized 
having a particular face pose; inputting a generic three dimensional face 
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model; creating a specific three dimensional face model of the specific 
subject sought to be recognized by deforming the generic face model to 
conform to the shape of the face depicted in the input image; 
synthesizing various face pose images using the specific 3-D face 
model: and employing the synthesized images as training images to 
train a recognizer . " (emphasis added) 

B. The 35 U.S.C. 5103(e) Rejection of Claims 12-15: 

Claims 12-15 were rejected under 35 USC 103(a) as unpatentable over Yan, in view 
of Kung, and in further view of Deering, U.S. Patent No. 6,525,723, hereinafter Deering. 
The Examiner stated that Yan and Kung do not teach assigning the color intensity of each 
pixel in order to perform textural analysis. However, the Examiner contended that Deering 
teaches this feature. The Examiner further contended it would have been obvious to have 
modified the Yan and Kung systems by Deering's color intensity calculations. The 
appellants respectfully traverse this contention of obviousness. 

The Appellants' claimed invention allows for face recognition even in the 
absence of a significant amount of training data. Further, it can recognize faces at 
various pose angles even without actual training images exhibiting the corresponding 
pose. This is accomplished by synthesizing training images depicting a subject's face 
at a variety of poses from a small number of actual images of the subject's face and 
using these synthesized images to train a recognizer. 

Neither Yan, Kung, nor Deering disclose a system and method that allows 
the training of a recognition system by employing at least one image to generate 
a synthesized training database. Nor does Yan in combination with Kung and 
Deering recognize the advantages of the appellants' claimed invention. Namely, 
neither Yan, Kung, nor Deering, alone or in combination, teach the appellants' claimed 
system and method of face recognition using a training database made of synthesized 
images. 
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Thus, the appellants have claimed elements not taught in the cited art and which 
have advantages not recognized therein. Accordingly, no prima facie case of 
obviousness has been established in accordance with the holding of In Re Fine. This 
lack of prima facie showing of obviousness means that the rejected claims are 
patentable under 35 USC 103 over Yan in combination with Kung and Deering. 

C. The 35 U.S.C. S1 03(e) Rejection of Claim 17: 

Claim 17 was rejected under 35 USC 103(a) as unpatentable over Yan, in view of 
Kung and in further view of Georghiades, June 1999. The Examiner stated that Yan and 
Kung do not teach varying illumination to produce synthesized images. However, the 
Examiner contended that Georghiades teaches this feature. The Examiner further 
contended it would have been obvious to have modified the Yan and Kung systems by 
Georghiades' teachings. The appellants respectfully traverse this contention of 
obviousness. 

As discussed previously, the appellants 1 claimed invention allows for face 
recognition even in the absence of a significant amount of training data. Further, it can 
recognize faces at various pose angles even without actual training images exhibiting 
the corresponding pose. This is accomplished by synthesizing training images 
depicting a subject's face at a variety of poses from a small number of actual images of 
the subject's face. These synthesized images are used to train a recognizer. 

Georghiades teaches an illumination-based method for synthesizing images that 
does not require any knowledge of light source directions or the establishment of point 
or line correspondences. However, like Yan and Kung, Georghiades does not 
disclose a system and method that allows the training of a recognition system to 
recognize a person based on a synthesized database. 

Therefore, Yan, Kung, nor Georghiades, alone or in combination, do not 
disclose a system and method that allows the training of a recognition system to 
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recognize a person based on a synthesized database. Nor does Yan in 
combination with Kung and Georghiades recognize the advantages of the appellants' 
claimed invention. Namely, neither Yan, Kung, nor Georghiades, alone or in 
combination, teach the appellants 1 claimed system and method of face recognition 
using a training database made of synthesized images generated from as little as one 
image. 

Thus, the appellants have claimed elements not taught in the cited art and which 
have advantages not recognized therein. Accordingly, no prima facie case of 
obviousness has been established in accordance with the holding of In Re Fine. This 
lack of prima facie showing of obviousness means that the rejected claim is patentable 
under 35 USC 103 over Yan in combination with Kung and Georghiades. 

X. SUMMARY 

For the foregoing reasons, it is respectfully submitted that the Examiner's 
rejection of Claims 1-17 was erroneous. As such, reversal of the Examiner's decision is 
respectfully requested at the earliest opportunity. 




Respectfully submitted 



Lyon & Harr, LLP 
300 Esplanade Drive, Suite 800 
Oxnard, California 93036 
(805) 278-8855 



Registration No. 42,821 
Attorney for Appellants 
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CLAIMS 



1. (Original) A computer-implemented process for face recognition, comprising 
using a computer to perform the following process actions: 

inputting an image of a face of a subject sought to be recognized having a 
particular face pose; 

inputting a generic three dimensional face model; 

creating a specific three dimensional face model of the specific subject sought to 
be recognized by deforming the generic face model to conform to the shape of the face 
depicted in the input image; 

synthesizing various face pose images using the specific 3-D face model; and 
employing the synthesized images as training images to train a recognizer. 

2. (Orginal) The computer-implemented process of Claim 1 further 
comprising the process action of using a spline surface construction technique to 
smooth the specific face model. 

3. (Original) The computer-implemented process of Claim 2 further 
comprising the process action of using a texture mapping technique to endow textural 
detail to the smoothed face model. 

4. (Original) The computer-implemented process of Claim 1 wherein the 
process action for inputting an image of a face comprises inputting one frontal view and 
one profile view of the face. 
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5. (Original) The computer-implemented process of Claim 1 wherein the 
process action for inputting an image comprises inputting at least two views of a face 
each having poses varying in orientation from the others by at least 15 degrees. 

6. (Original) The computer-implemented process of Claim 1 wherein the 
process action for inputting a generic three dimensional face model comprises 
employing a polygon model that depicts the surfaces of the face as a series of vertices 
defining a facial mesh. 

7. (Original) The computer-implemented process of Claim 6 wherein the 
process action for creating a specific three dimensional face model of the subject 
comprising using a deformation technique to create a specific three dimensional model 
of the subject from the generic model and the input images. 

8. (Original) The computer-implemented process of Claim 7 wherein the 
process action of using a deformation technique to create a specific three dimensional 
model further comprises the process actions of: 

extracting feature point sets from input frontal and profile images of the 

subject; and 

modifying the generic face model to the specific face model by 
comparison and mapping between the feature point sets. 
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9. (Original) The computer-implemented process of Claim 4 wherein 
symmetry of the face is assumed, such that for the purposes of said creating of a 
specific 3-D face model it is assumed the side of the face opposite that depicted in the 
profile image has the same shape and coloring. 

10. (Original) The computer-implemented process of Claim 1 wherein two 
or more images are used to create the model of a specific face, further comprising a 
process action of refining the specific model of the face by using additional images. 

1 1 . (Previously Presented) The computer-implemented process of Claim 2 
wherein a facet-based representation of the specific face is replaced with a spline 
surface representation. 

12. (Original) The computer-implemented process of Claim 3 wherein the 
process action to endow textural detail to the face model further comprises the process 
action of using the input images to assign color intensity to each pixel of the three 
dimensional face model. 

13. (Original) The computer-implemented process of Claim 12 wherein a 
Bezier patch technique is employed to add texture detail to a face model. 

14. (Original) The computer-implemented process of Claim 1 3 wherein the 
process action for assigning color intensity further comprises the process action of: 
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for each Bezier surface patch of the face surface determining a 
corresponding texture patch by mapping the boundary curve of the Bezier patch to the 
face image. 

1 5. (Original) The computer-implemented process of Claim 1 3 wherein 
both frontal and profile images are employed and wherein the face image chosen to 
provide the texture information depends on the preferred direction of the Bezier patch; 
such that the frontal image is used to map when the angle between the direction and 
the Y-Z plane is less than 30 degrees and wherein a profile image is used when the 
angle between the direction and the Y-Z plane is greater than 30 degrees. 

16. (Original) The computer-implemented process of Claim 1 wherein face 
pose images are synthesized for every in-plane rotation of plus or minus 10 to 15 
degrees and every out-of-plane rotation of plus or minus 15-20 degrees, with 
increments of about 3-7 degrees within a group. 

17. (Previously presented) The computer-implemented process of Claim 1 
wherein illumination is varied to produce the synthetic images at varied illuminations by 
taking the generated images and synthetically varying the illumination to produce each 
image at various illuminations. 

18. (Original) A system for generating synthesized face images, the 
system comprising: 
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at least one camera positioned so capture at least one image of a subject 
for whom images are to be synthesized; 

a general purpose computing device; and 

a computer program comprising program modules executable by the 
computing device, wherein the computing device is directed by the program modules of 
the computer program to, 

input at least one image of a face of a subject from the at least one 

camera; 

input a generic face model; 

create a specific face model of the subject by deforming the generic 
face model to conform to the shape of the face depicted in the input image; and 

synthesize various face poses using the specific 3-D face model. 

19. (Original) The system of Claim 18, further comprising a computer 
module to, 

employ the synthesized images as training images by inputting the 
synthesized images into a recognizer. 

21. (Original) The system of Claim 18, further comprising a computer 
module to, 

use a spline surface technique to smooth the specific face model. 

22. (Original) The system of Claim 21 , further comprising a computer 
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module to, 



use a texture mapping technique to endow textural detail to the 



smoothed face model. 

23. (Original) A computer-readable medium having computer-executable 
instructions for generating synthesized images of a face, said computer-executable 
instructions comprising: 

inputting an image of a face of a subject having a practicular face pose; 

inputting a generic three dimensional face model; 

creating a specific three dimensional face model of the specific subject by 
deforming the generic face model to conform to the shape of the face depicted in the 
input image; 

using a spline surface construction technique to smooth the specific face model; 
using a texture mapping technique to endow textural detail to the smoothed face 

model; 

synthesizing various face pose images using the specific 3-D face model; and 
employing the synthesized images as training images to train a recognizer. 



Respectfully submitted 



Lyon & Harr, LLP 
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